Projects
️📏 Job Satisfaction Measurement using LLM-embeddings
Date: January 16, 2025
Traditional Likert scales, the preferred choice for many psychology researchers, come with notable limitations: predefined response options that constrain responses, careless responding, debates about whether response categories are truly equidistant, and susceptibility to common method bias, which can inflate relationships. In contrast, natural language, expressed through open text, is how we naturally communicate. What if we could leverage this to measure psychological constructs?
In my master’s thesis, I explore the use of BERT-based LLM embeddings to quantify text responses and measure job attitudes, with a focus on job satisfaction. I evaluate the effectiveness of these embeddings by examining both their construct validity and criterion validity in assessing job satisfaction.
Figure 1: Correlation between LLM-predicted and actual job satisfaction
Figure 2: Predicting Turnover Intention using LLM-embeddings
Additionally, I conducted exploratory analyses to provide practical insights into implementing LLMs. These analyses identified the optimal variations of BERT and hidden layers for accurately measuring job-related psychological attitudes. For a detailed breakdown of the embedding process and how these embeddings are fed into machine learning models, check out my blog post titled “From Text to Predictions”.
💼 Adverse Impact Analysis
Date: December 22, 2024
In line with EEOC Uniform Guidelines, hiring practices can be discriminatory due to disparate treatment (intentional) or adverse impact (unintentional). Adverse impact arises when the selection ratio for protected groups is less than four-fifths of that for the majority group (Spector, 2020). Thus, it is crucial for companies to measure these critical metrics and take necessary steps to adjust their assessments if adverse impact is detected.
For this project, I conducted an adverse impact analysis to evaluate whether an assessment used for management candidates disproportionately affects protected groups.
Additionally, I analyzed dimension scores across positions held by applicants and examined whether passing rates varied by month.
🕰️ Delay Discounting as a Latent Factor
Date: December 5, 2024
This project examines the latent factor structure of delay discounting, the tendency to prioritize immediate rewards over delayed ones, which is linked to behavioral outcomes such as substance abuse gambling, credit card debt, and poor academic performance (Wölfling et al., 2020). Previous studies have used various methods to measure delay discounting, but the findings have been inconsistent, partly due to differences in operationalization. This study uses confirmatory factor analysis (CFA) to explore the underlying latent factors of delay discounting and their relationship to behavioral outcomes, providing a clearer understanding of the construct and its implications.
One Factor Model:
CFI = .72
RMSEA = .24
SRMR = .109
Avg R2 = .60
Two Factor Model:
CFI = .94
RMSEA = .12
SRMR = .04
Avg R2 = .69
Four Factor Model:
CFI = .96
RMSEA = .10
SRMR = .04
Avg R2 = .69
🤖 Which ML Algorithms Predict Job Satisfaction The Best?
Date: May 2, 2023
Machine learning algorithms have gained significant popularity in I/O psychology due to their advanced learning capabilities, often outperforming traditional regression methods in predictive tasks. However, their “black-box” nature remains a challenge for research justification. This project compares the performance of baseline model logistic regression with popular algorithms KNN, and random forest in a 4-class job satisfaction classification task using the IBM HR dataset from Kaggle, comprising approximately 23,000 observations. Using lasso-based feature-selection methods, hyperparameter tuning, the project optimizes model performance and identifies the algorithm with the highest predictive accuracy. The findings offer actionable insights into employee well-being, showcasing the potential of data-driven approaches to enhance workforce engagement and organizational performance.
⚖️ Conscientiousness Scale: Development & Validation
Date: April 21, 2023
This project involved the psychometric development of a new Conscientiousness scale, one of the Big Five personality traits. Following best-practice item-writing guidelines, I conducted a pilot study and refined the item pool by removing items with low item-total correlations and minimal impact on Cronbach’s alpha if removed (see Figure 1 & 2). Subsequent analyses demonstrated strong internal consistency (α = .91) and validity evidence. The new scale exhibited high convergent validity (r = .85) with the well-validated IPIP Conscientiousness scale and good discriminant validity with other Big Five dimensions (see Figure 3). Criterion validity was supported by a positive correlation with job performance (r = .33), consistent with meta-analytic findings (Sackett et al., 2022), establishing the scale as a valid measure of conscientiousness.
Figure 1
Figure 2
Figure 3